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Description 

METHOD AND APPARATUS OF 
MUL TI- MODALITY IMA GE FUSION 

Background of Invention 

[0001] The present invention relates generally to diagnostic 

imaging and, more particularly, to a method and appara- 
tus of combining or fusing functional diagnostic data and 
anatomical diagnostic data acquired of a subject with 
imaging systems of different modalities to generate a 
composite image for clinical inspection. 

[0002] The fusion of functional image data and anatomical image 
data is a widely-practiced technique to provide composite 
images for improved pathology identification and clinical 
diagnosis. Typically, the functional and anatomical image 
data is acquired using nuclear medicine based systems 
such as single-photon computed tomography (SPECT), 
and positron emission tomography (PET) or radiology 
based imaging systems such as computed tomography 
(CT), magnetic resonance (MR), ultrasound, and x-ray. 



Generally, it is desirable to "fuse" an image from SPECT or 
PET with an image from CT or MR. In this regard, it is typ- 
ically desired for the functional image from SPECT or PET 
to be superimposed on the anatomical image acquired us- 
ing CT or MR. 

[0003] Fusion of functional and anatomical data that has been 
acquired separately with imaging systems predicated on 
different imaging technologies can be problematic. That 
is, the functional data may be acquired at a different time 
than the anatomical data. As such, patient positioning be- 
tween the separate data acquisitions typically varies. Dif- 
ferent size acquisitions with different slice thickness and 
pixel sizes with different central points are also not un- 
common. As such, for a clinically valuable composite im- 
age to be produced, these differences as well as others 
typically encountered, must be resolved. 

[0004] one solution has been the development of a hybrid scan- 
ner capable of acquiring PET and CT images during a sin- 
gle scan study in such a manner to avoid many of the 
drawbacks enumerated above. A combined PET/CT scan- 
ner, however, may not be feasible in all circumstances. For 
instance, it may not be practical for a diagnostic imaging 
center, hospital, or the like to replace existing PET and CT 



systems with a combined imager. Moreover, a combined 
PET/CT scanner, by definition, may generate a composite 
image of functional and anatomical data acquired using 
PET and CT, respectively. However, the scanner cannot 
provide a composite image of PET and MR data, SPECT and 
MR data, or SPECT and CT data. As such, a hybrid system 
may not address the myriad of diagnostic needs required 
of a radiologist or other health care provider in rendering 
a diagnosis to a patient. 
[0005] Another solution that is consistent with conventional fu- 
sion techniques fails to adequately address the drawbacks 
associated with the overlaying of collocated functional and 
anatomical data that are not registered. That is, present 
fusion protocols combine data having a common coordi- 
nate alignment, but fail to register the functional and 
anatomical images. Registering is commonly defined as 
the process of aligning medical image data. This is based 
on the premise that the functional and anatomical data 
sets were acquired under identical physiological states 
and therefore can be fused without taking additional mea- 
sures into account. In this regard, conventional fusion 
techniques orientate the functional and anatomical data 
but do not take measures to sufficiently align the func- 



tional and anatomical data. Furthermore, the image reso- 
lution from PET and SPECT are limited by maximum en- 
ergy resolution of positron-emitting isotopes. The resolu- 
tion of functional images compared to anatomical images 
is notably inferior. Another consideration that specifically 
affects cardiac imaging is the considerable amount of mo- 
tion that can add additional blurring to any image set. The 
goal of anatomical imaging in the heart is to observe the 
heart without motion. Functional imaging of the heart can 
compensate for motion by dividing the imaging into bins 
but the number of bins is the denominator when the total 
dataset is the numerator. The number of coincidence 
events is limited to the number of radioactive decay 
events and being able to observe as much data as possi- 
ble is desirable for a successful diagnosis. As a result, the 
radiologist or other health care provider must decipher a 
single composite image with the functional and anatomi- 
cal information, with respect to one another, being mis- 
aligned. Additional post-fusion processing steps may be 
taken to correct the misalignment of the respective im- 
ages. 

[0006] a conventional fusion of CT and PET image data is illus- 
trative of the above drawbacks. During a PET/CT cardiac 



acquisition, the CT study is performed with ECG gating 
and the PET study may or may not be performed with ECG 
gating. The anatomical position of the heart typically 
changes relative to the ECG cycle. During image process- 
ing the CT image is reconstructed from a portion of the 
data centered on a selected phase during the cardiac cycle 
in order to provide an image with the least amount of mo- 
tion blurring artifacts. The CT coronary arteries are then 
tracked and segmented out of the CT image. The seg- 
mented images retain the coordinate system of the origi- 
nal data frozen at one particular phase of the cardiac cy- 
cle. A static or dynamic PET image may then be recon- 
structed from the entire set of PET data that is averaged 
over many ECG cycles. A gated PET image set is recon- 
structed for each bin in the gated study. One of these bins 
may correspond to the selected phase for which the CT 
data set was reconstructed. The alignment may further 
improve with such conditions. These PET images are then 
processed such that the left ventricle is segmented based 
on the long axis of the heart. Using this information, a 
PET 3D model can be displayed in "model" space that ap- 
proximates the anatomical shape of a left ventricle. The 
CT image is then fused with the PET image along the 



model coordinates to form a composite image. However, 
the respective images from which the composite image is 
formed are not registered because the coordinate systems 
are not common to both image sets. Depending on the 
amount of image blurring due to radioactive tracer en- 
ergy, degree of cardiac motion, and the modeling tech- 
niques, different amounts of misalignment may be intro- 
duced. As such, the composite image typically must un- 
dergo additional and time-consuming processing to ef- 
fectively align the functional data with the anatomical data 
in a clinical area of interest to provide optimal images for 
diagnosis. 

[0007] Another classic multi-modality paradigm aligns internal or 
external fiducial markers from a functional image with 
corresponding anatomical points on an anatomical image. 
This conventional fiducial marker-based system imple- 
ments a manual method of fusion that does not take local 
variations in the datasets into account. The conventional 
automated rigid or non-rigid body registration process 
uses mutual information as the cost function for high- 
lighting differences between the functional and anatomi- 
cal images. The cost function therefore defines or guides 
the registration process of the functional data to the 



anatomical data. There are also methods that use fiducial 
markers and rigid and non-rigid affine transformation to 
register images. However, these automated methods do 
not use any localized anatomical constraints to guide 
them. As a result, these conventional approaches may 
only perform data-to-data fusion and, as such, are inap- 
plicable when fusion between data and modeled data, or 
fusion between modeled data and modeled data is de- 
sired. 

[0008] Therefore, it would be desirable to design an apparatus 
and method of fusing multi-modality images such that 
alignment is resolved prior to the fusion of the separate 
images such that post-fusion processing is reduced and 
supports fusion of modeled functional and/or anatomical 
data. 

Brief Description of Invention 

[0009] The present invention is directed to a method and appa- 
ratus for fusing or combining functional image data and 
anatomical image data that overcome the aforementioned 
drawbacks. The invention, which may be carried out 
through user interaction or automatically, enables com- 
posite and clinically valuable images to be generated that 
display functional and anatomical data acquired with dif- 



ferent imaging systems. By identifying fiducial markers on 
a functional data image and correlating the fiducial mark- 
ers with anatomical markers or indicia on the anatomical 
data image, the respective images may be aligned with 
one another before a composite image is generated. 
Warping is carried out that takes into consideration 
anatomical constraints while maintaining alignment of the 
fiducial and anatomical markers. 

[0010] Therefore, in accordance with one aspect of the invention, 
a method of medical image overlap comprises the steps of 
determining at least two anatomical fiducial markers on a 
functional image and determining corresponding points to 
the at least two anatomical fiducial markers on an 
anatomical image. The method also includes the step of 
aligning the at least two anatomical fiducial markers with 
the corresponding points on the anatomical image and 
the step of warping the functional image to fit constraints 
of the anatomical image while maintaining alignment of 
the at least two anatomical fiducial markers and the cor- 
responding points on the anatomical image. 

[001 1] According to another aspect of the invention, a diagnostic 
image generation system includes at least one database 
containing functional and anatomical image data and a 



computer programmed to determine at least a pair of 
fiducial markers on a functional image. The computer is 
also programmed to locate corresponding anatomical in- 
dicia on an anatomical image and generate a composite 
image of the functional image and the anatomical image 
such that the fiducial markers and the anatomical indicia 
are aligned and anatomical constraints are observed. 

[0012] | n accordance with yet another aspect of the present in- 
vention, a computer readable storage medium has a com- 
puter program stored thereon. The computer program 
represents a set of instructions that when executed by a 
computer cause the computer to access functional image 
data of a medical patient as well as anatomical image data 
of the medical patient. The computer is then programmed 
to identify more than one fiducial marker in the functional 
image data and identify anatomical locations in the 
anatomical image data that correspond to the more than 
one fiducial marker. The set of instructions further cause 
the computer to generate an image with the functional 
image data superimposed on the anatomical image data 
that considers anatomical constraints. 

[0013] Various other features, objects and advantages of the 

present invention will be made apparent from the follow- 



ing detailed description and the drawings. 
Brief Description of Drawings 



[0014] The drawings illustrate one preferred embodiment 

presently contemplated for carrying out the invention. 
[0015] in the drawings: 

[0016] pig. 1 is a schematic representation of a multi-node net- 
work of medical imaging systems applicable with the 
present invention. 

[0017] pig. 2 is a flow chart setting forth the steps of a functional 

image data and anatomical image data fusion technique in 

accordance with the present invention. 
Detailed Description 

[001 8] j ne present invention will be described with respect to a process, 
which may be carried out through interaction with a user or auto- 
matically, to generate a composite diagnostic image of functional 
and anatomical data acquired separately with a PET imaging sys- 
tem and a CT imaging system. One skilled in the art will appreciate, 
however, that imaging systems of other modalities such as MR, 
SPECT, ultrasound, x-ray, and the like may be used to acquire the 
functional and anatomical data to be combined into a composite 
image. Further, the present invention will be described with respect 
to the acquisition and imaging of data from a cardiac region of a 



patient. However, one skilled in the art will appreciate that the 
present invention is equivalently applicable with data acquisition 
and imaging of other anatomical regions of a patient. 
[0019] Referring now to Fig. 1, an overview block diagram of a 
medical diagnostic and service networked system 10 is 
shown which includes a plurality of remote treatment sta- 
tions, such as Station A referenced with numeral 12, and 
Station B referenced with numeral 14, which may include a 
medical treatment facility, hospital, clinic, or mobile 
imaging facility. It is understood, that the number of 
treatment stations can be limitless, but two specific em- 
bodiments are shown with Station A and Station B, which 
will be further explained hereinafter. The treatment sta- 
tions 12, 14 are connected to a centralized facility 16 
through a communications link 18, such as a network of 
interconnected server nodes. This network of intercon- 
nected nodes may be a secure, internal, intranet or a pub- 
lic communications network, such as the internet. Al- 
though a single centralized facility is shown and de- 
scribed, it is understood that the present invention con- 
templates the use of multiple centralized facilities, each 
capable of communication with each treatment station. 
Each treatment station has operational software associ- 



ated therewith which can be serviced by the centralized 
facility 16. 

[0020] The various systems disclosed are configured to be selec- 
tively linked to the centralized facility 16 by a workstation, 
which in the example of treatment station 12, includes a 
laptop computer 20 or permanent workstation 26 con- 
nected to an internal network 22. Such selective linking is 
desirable for accessing data from the systems and trans- 
mitting data to the systems. 

[0021] | n general, a treatment site may have a number of devices 
such as a variety of medical diagnostic systems of various 
modalities. As another example, in the present embodi- 
ment, the devices may include a number of networked 
medical image scanners 24 connected to the internal net- 
work 22. Alternately, a treatment station or treatment site 
14 can include a number of non-networked medical im- 
age scanners 28, 30, and 32 each having a computer or 
work station associated therewith and having an internal 
modem or network connection device 34, 36, and 38 to 
connect the remote treatment station to a communica- 
tions link 18, such as the internet, to communicate with 
centralized facility 16. 

[0022] it j S understood that each of the network scanners 24 has 



its own workstation for individual operation and are linked 
together by the internal network 22. Additionally, each of 
the network/scanners may be linked to a local database 
40 configured to store data associated with imaging scan 
sessions, as will be discussed shortly. Further, such a sys- 
tem is provided with communications components allow- 
ing it to send and receive data over a communications link 
18. Similarly, for the non-networked medical image scan- 
ners at remote treatment station 14, each of the scanners 
28, 30, and 32 is connected to communications link 18 
through which they can communicate with the centralized 
facility 16. Furthermore, each scanner 28, 30, 32 may in- 
clude a database 42, 44, 46, respectively, for storing 
scanning data. Scanning data may be transferred to a cen- 
tralized database 48 through communications link 18 and 
router 50. The centralized database 48 is included in a re- 
mote file server 52, where workstations and scanners, ex- 
ternal to the local intranet containing the centralized 
database 48, can access the database as though located 
locally on the intranet 54. More specifically, as will be de- 
scribed, workstations 20, 26 can access the data stored in 
the centralized database 48, or other remote database, 
such as database 40, as though the data were stored in a 



database within the specific workstation requesting the 
data. 

[0023] The embodiment shown in Fig. 1 contemplates a medical 
facility having such systems as MRI systems, ultrasound 
systems, x-ray systems, CT systems, as well as PET sys- 
tems, nuclear imaging systems, or any other type of med- 
ical imaging system, however, the present invention is not 
so limited. Such facilities may also provide services to 
centralized medical diagnostic management systems, pic- 
ture archiving and communications systems (PACS), tel- 
eradiology systems, etc. Such systems can be either sta- 
tionary and located in a fixed place and available by a 
known network address, or be mobile having various net- 
work addresses. In the embodiment shown in Fig. 1, each 
treatment station 12, 14 can include any combination of 
the aforementioned systems, or a treatment station may 
have all of a single type of system. Each system is con- 
nectable and can transmit data through a network and/or 
with at least one database 40, 48. However, it is under- 
stood that the single representation of the centralized 
database 48 is for demonstrative purposes only, and it is 
assumed that there is a need for multiple databases in 
such a system. 



[0024] As previously discussed, each of the systems and substa- 
tions described herein and referenced in Fig. 1 may be 
linked selectively to the centralized facility 16 via a net- 
work 18. According to the present invention, any accept- 
able network may be employed whether public, open, 
dedicated, private, or so forth. The communications links 
to the network may be of any acceptable type, including 
conventional telephone lines, fiber optics, cable modem 
links, digital subscriber lines, wireless data transfer sys- 
tems, or the like. Each of the systems is provided with 
communications interface hardware and software of gen- 
erally known design, permitting them to establish network 
links and exchange data with the centralized facility 16. 
However, the systems or particularly, workstations 20, 26 
are provided with specialized software so as to communi- 
cate with the centralized facility 16 and particularly with 
the remote database 48 as though the data stored in the 
remote database is located locally on workstation 20. In 
some cases, during periods when no data is exchanged 
between the customer stations and the centralized facility, 
the network connection can be terminated. In other cases, 
the network connection is maintained continuously. 

[0025] | n one embodiment, the scanning data from as imaging 



session, for example, on scanner 24, is automatically 
transmitted from the scanner to the database 48. That is, 
database 48 is automatically updated after each imaging 
scan is executed. Records must be maintained as to the 
dosage used and catalogued according to the particular 
diagnostic procedure as well as the individual patient. 
From these records, the treatment facilities or institutions 
may ensure conformity with dosage guidelines and regu- 
lations. Further, as a result of maintaining an active 
database storing scan parameter values of executed 
imaging sessions, a user or prescriber of an imminent 
imaging session may query the database to later retrieve 
scanning data for review from any workstation 20, 26 that 
is permitted to access the remote database 48. 
[0026] As described above, the database having the scan param- 
eter values stored thereon may be accessed from a num- 
ber of scanners that are remotely located from the 
database. Furthermore, there is no requirement that each 
scanner be physically located in the same treatment sta- 
tion or facility. That is, a scanner located in station 12 
may electronically transmit and receive data from the re- 
mote database 48 while simultaneously therewith any 
scanner 28, 30, 32 in station 14 may likewise transmit 



and receive data to and from database 48. Later a work- 
station 20, 26 at any locality, for example that may be re- 
mote to both the scanner 24 and the centralized facility 
16, can access the data from any scanner 24, 28, 30, 32 
by accessing the centralized facility 16. Furthermore, 
database 48 need not be located in a separate centralized 
facility 16. That is, database 48 may be located in either 
one of stations 12, 14 as well as be remotely located 
within that station or treatment facility and the worksta- 
tion 20, 26 requiring access to the scanning data. 

[0027] Referring now to Fig. 2, the steps of a processing technique or 
method for aligning and registering functional and anatomical 
data acquired from separate imaging systems built on separate 
imaging technologies are set forth. The process may be automated 
or guided through user interactions and commands. 

[0028] process 56 begins with the accessing of anatomical image 
data 58 and functional image data or a model of func- 
tional image data 60. A model of functional image data 
may be defined as segmented image data with arbitrary or 
similar intensities as the original functional image data 
from which the model was generated. Arrow 62 indicates 
that the anatomical image data and the functional image 
data are geometrically collocated. That is, the anatomical 



and functional data are geometrically oriented about a 
common coordinate system; however, the data are not 
registered. 

[0029] process 56 continues with the identification of anatomical 
fiducial markers on the functional image 64. Preferably, at 
least two reference markers are identified. The reference 
markers, as will be described below, are used to identify 
corresponding anatomical locations on the anatomical im- 
age. Additionally, it is preferred that the fiducial markers 
be internal anatomical features. However, external surface 
markers may be used, but the external markers must be 
used during the acquisition of the functional data as well 
as the anatomical data. This may be problematic given 
that the functional data may be acquired at a different 
time and location than the acquisition of the anatomical 
data. For example, in a cardiac study, the reference mark- 
ers may include the ventricular grooves between respec- 
tive ventricles of a patient's heart. 

[0030] Following determination and identification of fiducial 

markers on the functional image, corresponding anatomi- 
cal indicia or points are determined 66 on the anatomical 
image. In the cardiac study example given above, the ven- 
tricular grooves would be identified on the anatomical im- 



age. Once the corresponding anatomical indicia are deter- 
mined and identified, the functional image is overlaid 68 
on the anatomical image such that anatomical indicia and 
the fiducial markers are cooperatively aligned. In this re- 
gard, in a preferred embodiment, the anatomical image 
remains fixed and the functional image is superimposed 
thereon. 

[0031] The aligning of the fiducial markers and the correspond- 
ing anatomical indicia may be carried automatically by a 
computer programmed to do so or may be done through 
user interaction with a graphical user interface (GUI) dis- 
playing each of the images. In this regard, the user, such a 
radiologist, technician, or health care professional, may 
electronically "grab" the functional image, "drag" the im- 
age over the anatomical image such that the fiducial 
markers and anatomical indicia are aligned, and "drop" 
the functional image on the anatomical image. In another 
embodiment, the user may identify or "highlight" the re- 
spective fiducial markers and anatomical indicia, and then 
instruct the computer to overlay or superimpose the func- 
tional image on the anatomical image. Additionally, to 
sufficiently align the fiducial markers and the correspond- 
ing anatomical indicia it may be necessary to carry out 



various translation, scaling, and rotation processes. 
[0032] Process 56 continues at step 70 with the warping of the 
functional data to the anatomical data such that anatomi- 
cal constraints are met while maintaining alignment of the 
fiducial markers and the corresponding anatomical indi- 
cia. In this regard, the process tailors the warping process 
to anatomical constraints of the anatomical data rather 
than a direct warping of the functional and anatomical 
data. For instance, in the cardiac example above, the 
health care provider will recognize that the functional data 
corresponds to ventricular anatomy and the anatomical 
data corresponds to the coronary artery. As it is common 
for the coronary arteries to be located on the outer sur- 
faces of the ventricles, warping would be applied locally 
such that the coronary arteries of the anatomical image 
lay on the outer surface of the ventricular anatomy of the 
functional image. In this case, enforcing the anatomical 
constraint requires that the nearest point on the ventricu- 
lar surface project onto the location of the coronary artery 
while maintaining a smooth surface. As a result, the func- 
tional and anatomical are more precisely aligned and the 
composite image generated at step 72 is clinically valu- 
able. As noted above, anatomical constraints are applica- 



tion and modality dependent and are useful for creating 
clinically meaningful results. In this invention, the 
anatomic constraints are used to define physical relation- 
ships between aspects covered by functional and 
anatomic data, and to enforce known relationships be- 
tween functional and anatomic data. 

[0033] Warping is an elastic registration process that may be 

used to fuse or combine images acquired from scanners 
of separate modalities. With warped, elastic transforma- 
tion techniques, multi-scale, multi-region, pyramidal ap- 
proaches are implemented. As such, a cost function is uti- 
lized to highlight differences between the images on a 
scale-by-scale basis such that the differences are opti- 
mized at every scale. That is, an image is sampled at a 
given scale and then is segmented or divided into multiple 
regions. Separate shift vectors are then determined or cal- 
culated at different regions. The vectors are interpolated 
to generate a smooth shift transformation which is ap- 
plied to warp the image. The image is then re-sampled 
and the registration process is repeated at successive 
scales until a pre-determined final scale is reached. 

[0034] The above process has been described with respect to the 
fusion of data between anatomical image data and either 



functional image data or modeled functional image data. 
The process, however, may be equivalently carried out to 
fuse modeled anatomical image data and either functional 
image data or modeled functional image data. 

[0035] Therefore, in accordance with one aspect of the invention, 
a method of medical image overlap comprises the steps of 
determining at least two anatomical fiducial markers on a 
functional image and determining corresponding points to 
the at least two anatomical fiducial markers on an 
anatomical image. The method also includes the step of 
aligning the at least two anatomical fiducial markers with 
the corresponding points on the anatomical image and 
the step of warping the functional image to fit constraints 
of the anatomical image while maintaining alignment of 
the at least two anatomical fiducial markers and the cor- 
responding points on the anatomical image. 

[0036] According to another aspect of the invention, a diagnostic 
image generation system includes at least one database 
containing functional and anatomical image data and a 
computer programmed to determine at least a pair of 
fiducial markers on a functional image. The computer is 
also programmed to locate corresponding anatomical in- 
dicia on an anatomical image and generate a composite 



image of the functional image and the anatomical image 
such that the fiducial markers and the anatomical indicia 
are aligned and anatomical constraints are observed. 

[0037] | n accordance with yet another aspect of the present in- 
vention, a computer readable storage medium has a com- 
puter program stored thereon. The computer program 
represents a set of instructions that when executed by a 
computer cause the computer to access functional image 
data of a medical patient as well as anatomical image data 
of the medical patient. The computer is then programmed 
to identify more than one fiducial marker in the functional 
image data and identify anatomical locations in the 
anatomical image data that correspond to the more than 
one fiducial marker. The set of instructions further cause 
the computer to generate an image with the functional 
image data superimposed on the anatomical image data 
that considers anatomical constraints. 

[0038] The present invention has been described in terms of the 
preferred embodiment, and it is recognized that equiva- 
lents, alternatives, and modifications, aside from those 
expressly stated, are possible and within the scope of the 
appending claims. 



